Image processing apparatus, image processing method, image processing program, and computer-readable record medium storing image processing program

ABSTRACT

An image processing apparatus has an edge direction estimation section for estimating edge direction information for a first image area containing a notable pixel, an edge pattern selection section for selecting an edge shape pattern corresponding to the first image area corresponding to a predetermined edge direction estimated based on the edge direction information and pixel values in the first image area containing the notable pixel, an image enhancement section for enhancing the pixel values of the first image area containing the notable pixel, an enlarged image block generation section for generating an enlarged image area using the selected edge shape pattern and the enhanced pixel values of the first image area, and an image block placement section for placing the enlarged image area generated by the enlarged image block generation section according to a predetermined method.

[0001] The present disclosure relates to the subject matter contained inJapanese Patent Application No. 2002-076896 filed on Mar. 19, 2002,which is incorporated herein by reference in its entirety.

BACKGROUND OF THE INVENTION

[0002] 1. Field of the Invention

[0003] This invention is intended for enlargement processing of an imagerepresented with a multi-step gradation and relates in particular to animage processing apparatus, an image processing method, an imageprocessing program, and a computer-readable record medium storing theimage processing program for suppressing occurrence of image qualitydefects of blurring and jaggies, in an input image and performingenlargement processing of the image with high quality.

[0004] 2. Description of the Related Art

[0005] Image enlargement processing is one type of basic processing fora system for editing, filing, displaying, and printing, images. Inrecent years, with the widespread of image data mainly intended fordisplay at a monitor display resolution to view an Internet web site ora digital video, high-quality enlargement processing has been becomingimportant increasingly to produce the high-quality output result inprinting the low-resolution images on a high-resolution printer.

[0006] Nearest neighbor method, linear interpolation or bilinear method,and cubic convolution method are available as existing techniques ofperforming enlargement processing of an image represented with amulti-step gradation, which will be hereinafter referred to as amultilevel image.

[0007] The nearest neighbor method is a method of using as each pixelvalue after enlargement, the pixel value of the nearest pixel when eachpixel is inversely mapped onto the original image. The nearest neighbormethod can be processed at high speed since it involves smallcomputation amount, but one pixel of the original image is enlargedintact to a rectangular shape. Thus, jaggies occur in an oblique lineportion or if the magnification is large, the image becomes mosaic; thedegree of image quality degradation is large.

[0008] The linear interpolation or bilinear method is a method ofassuming that the pixel value between pixels changes linearly, andlinearly interpolating the pixel values of four nearby pixels of theinverse map point of pixel after enlargement to find the pixel value.The linear interpolation or bilinear method involves heavier processingload than the nearest neighbor, but involves comparatively smallcomputation amount and jaggies occur less. On the other hand, the linearinterpolation or bilinear method has disadvantage in that the wholeimage becomes blurred centering on the edge portions not applied to theassumption of linear change.

[0009] The cubic convolution method is a method of defining aninterpolation function approximating a sinc function (sin(x)/x) based ona sampling theorem and performing convolution of 16 nearby pixels of theinverse map point of pixel after enlargement (four by four pixels in theX and Y direction) and the approximate interpolation function to findthe pixel value after enlargement. This method provides comparativelygood image quality as compared with the two techniques described above,but involves a large reference range and large computation amount andhas a characteristic of high-frequency enhancement. Thus, the cubicconvolution has disadvantages in that light jaggies occur in the edgeportions and the noise component is enhanced.

[0010] As attempts to solve the problems, techniques are proposed inJP-A-7-182503, JP-A-2000-228723, JP-A-2000-253238, “Tekioutekina nijigenhyouhonka kansuu niyoru kouhinshituna gazoukakudaisaikousei” (GazouDenshi Gakkaishi Vol.28 No.5 P.P. 620-626).

[0011] In JP-A-7-182503, the maximum value and the minimum value aredetected from pixels of an N×M area (for example, 3×3) surrounding anotable pixel and further contrast and an intermediate value arecalculated and based on the contrast, either the maximum value or theminimum value and an average value of other values are derived asrepresentative values. Next, linear interpolation or bilinear processingof the notable pixel to N×M pixels is performed, binarization isexecuted with the calculated intermediate value as a threshold value,and the two representative values are placed according to the result ofthe binarization. If the contrast is small, the pixel values of thelinear interpolation or bilinear processing are used intact, whereby afinal enlarged image is provided.

[0012] Accordingly, no jaggies occur and good conversion with nointerpolation blurring even in a natural image is possible. However,there is a problem that the edge parts are determined based only on thecontrast and that it is difficult to reproduce edges directions. Thereis also a problem that block-like distortion occurs in the edge partsince the N×M area is made up of two values only.

[0013] In JP-A-2000-228723, an original image is binarized, and from thebinary image, the direction of an oblique component contained in theoriginal image is found by making a match determination with a providedtwo-dimensional pattern (matrix data), and interpolation processing isfound along the found oblique direction. In other portions, linearinterpolation or bilinear processing is performed.

[0014] Since the original image is binarized according to apredetermined threshold value before the direction determination of theoblique component is made, the method is effective for edges with largedensity difference, but involves a problem in reproducing edges withsmall density difference.

[0015] In JP-A-2000-253238, linear interpolation or bilinear processingis performed for an original image to enlarge the original image and inparallel, each surrounding N×M area containing a notable pixel in theoriginal image is binarized, and a match determination with a presetoblique line detection pattern is made. If the notable pixel belongs toan edge area (matches the oblique line detection pattern), its directioninformation is mapped to the notable pixel. Further, the directioninformation is mapped to the enlarged image provided by performing thelinear interpolation or bilinear processing, and smoothing filteringalong the direction of the mapped direction information is performed,whereby enlargement processing with jaggy occurrence suppressed isperformed.

[0016] However, the technique described in JP-A-2000-253238 is asequence of enlargement of the original image, pattern matching,enlargement of the direction information, and direction-dependentsmoothing filtering for the enlarged image, so that the techniqueinvolves a problem of the processing load becoming very large as theimage size becomes large.

[0017] In the cubic convolution, an error is caused by using anapproximation function with the sinc function discarded finitely becausethe sinc function is an infinite series, and the sampling theorem aimsat continuous and differentiable signals; whereas the art described in“Tekioutekina nijigen hyouhonka kansuu niyoru kouhinshitunagazoukakudaisaikousei” (Gazou Denshi Gakkaishi Vol.28 No.5 P.P. 620-626)introduces the fact that an image contains a large number ofdiscontinuous points as a problem, uses a function fitted fordiscontinuity and having locality, different from the sinc function asan interpolation function, detects the edge direction globally, anddeforming the interpolation function in the detected edge direction,thereby providing an enlarged image with less occurrence of blurring andjaggies.

[0018] The function used in the technique is local, but the convolutionmatrix size becomes 4n relative to magnification n and the edgedirection is detected globally and thus there is a problem of a largecomputation amount.

SUMMARY OF THE INVENTION

[0019] It is therefore an object of the invention to provide an imageprocessing apparatus, an image processing method, an image processingprogram, and a computer-readable record medium storing the imageprocessing program for making it possible to provide an enlarged imagewith less occurrence of blurring and jaggies under small processing loadat high speed.

[0020] According to the invention, edge direction information isestimated for a first image area containing a notable pixel, an edgeshape pattern corresponding to the first image area corresponding to apredetermined edge direction estimated is selected based on the edgedirection information and pixel values in the first image areacontaining the notable pixel, and the pixel values of the first imagearea containing the notable pixel are enhanced. An enlarged image areais generated using the selected edge shape pattern and the enhancedpixel values of the first image area, and the enlarged image area isplaced according to a predetermined method.

[0021] In the invention, edge direction information is estimated for thefirst image area containing the notable pixel, an edge shape patterncorresponding to the first image area corresponding to a predeterminededge direction estimated is selected based on the edge directioninformation and the pixel values in the first image area containing thenotable pixel, so that it is made possible to determine the edge shapepattern corresponding to the edge direction only by executing simplepattern matching. Further, the pixel values of the first image areacontaining the notable pixel are enhanced, an enlarged image area isgenerated using the selected edge shape pattern and the enhanced pixelvalues of the first image area, and the enlarged image area is placedaccording to the predetermined method, so that it is made possible togenerate an enlarged image considering the edge direction and provide anenlarged image with less occurrence of blurring and jaggies under smallprocessing load at high speed.

BRIEF DESCRIPTION OF THE DRAWINGS

[0022]FIG. 1 is a block diagram to show the basic configuration of anembodiment of the invention;

[0023]FIG. 2 is a flowchart to show a processing flow of the embodimentof the invention;

[0024]FIGS. 3A and 3B are drawings to show examples of image blocksextracted by an image block setting section and notable areas andperipheral areas;

[0025]FIGS. 4A and 4B are drawings to describe specific examples ofnotable areas and peripheral areas and edge directions in notable area;

[0026]FIG. 5 is a flowchart to show a flow of edge direction estimationprocessing of an edge direction estimation section;

[0027]FIGS. 6A to 6C are drawings to show examples of reference areasused for edge direction estimation;

[0028]FIG. 7 is a flowchart to show a flow of edge direction estimationprocessing for an RGB color space color image in the edge directionestimation section;

[0029]FIG. 8 is a drawing to show an estimated edge direction in thenotable area in FIG. 4;

[0030]FIG. 9 is a drawing to show a specific example of an edge patterntable;

[0031]FIGS. 10A to 10E are drawings to specifically describe a method ofselecting a first edge pattern for the notable area in FIG. 4;

[0032]FIG. 11 is a drawing to show a second edge pattern in the notablearea in FIG. 4;

[0033]FIGS. 12A to 12C are drawings to show specific examples ofenhancement kernels used with an image enhancement section;

[0034]FIG. 13 is a drawing to show an example of enhancing pixel P withthe enhancement kernel in FIG. 12A;

[0035]FIG. 14 is a drawing to describe contrast enhancement using theenhancement kernels to enlarge input image data eight times;

[0036]FIG. 15 is a flowchart to show a flow of enlarged image blockgeneration processing of an enlarged image block generation section;

[0037]FIG. 16 is a drawing to show a specific example of 3×3 enlargedimage block generation; FIG. 17 is a drawing to show an example of 3×3enlarged image block generation using a second edge pattern differentfrom the second edge pattern in FIG. 16;

[0038]FIG. 18 is a drawing to show a specific example of 3×4 enlargedimage block generation;

[0039]FIG. 19 is a drawing to describe selection of reference pixelsaccording to estimated edge direction;

[0040]FIG. 20 is a drawing to show a specific example of 4×4 enlargedimage block generation;

[0041]FIG. 21 is a drawing to show an example of 4×4 enlarged imageblock generation when estimated edge direction is direction 0 or 4;

[0042]FIG. 22 is a drawing to show enlarged image block for notablearea; and

[0043]FIG. 23 is a drawing to show a specific example of 4×4 enlargedimage block placement in an image block placement section.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0044] Referring now to the accompanying drawings, there are shownpreferred embodiments of the invention. FIG. 1 is a block diagram toshow a processing section for performing enlargement processing of animage processing apparatus according to an embodiment of the invention.In the figure, an enlargement processing section 1 is made up of animage block setting section 10, an edge direction estimation section 11,an image enhancement section 12, an edge pattern selection section 13,an enlarged image block generation section 14, an image block placementsection 15, and an image data storage section 16. An outline of thecomponents of the embodiment and the operation thereof will bediscussed.

[0045] In FIG. 1, image data is described in an image format that can beprocessed in the image processing apparatus (for example, BMP, TIFF,PNG) and is generated from image data prepared in an application programfor performing processing of creation and editing, in a image processingmachines such as personal computer, or a digital camera (not shown).

[0046] The image data storage section 16 comprises a function oftemporarily storing image data input from an input unit (not shown)until the image data undergoes enlargement processing in the enlargementprocessing section 1, a function of temporarily storing enlarged imagedata subjected to resolution conversion or enlargement processing untilthe enlarged image data is output to an output unit (not shown).

[0047] The image block setting section 10 comprises a function ofsetting each block size required for processing of the edge directionestimation section 11 and the image enhancement section 12, extractingimage blocks of the block size in order from the input image data storedin the image data storage section 16, and transmitting the image blockto the edge direction estimation section 11 and the image enhancementsection 12.

[0048] The edge direction estimation section 11 comprises a function ofcalculating edge directions in a notable area in each of the imageblocks extracted in order by the image block setting section 10 and areference area on the periphery of the notable area from the pixel valuedistribution of each area and estimating the edge direction of thenotable area from the edge directions calculated in the areas.

[0049] The image enhancement section 12 comprises a function ofenhancing the contrast of the image data in the notable area in each ofthe image blocks extracted by the image block setting section 10 and theperipheral area of the notable area using the kernel of elements and thesize responsive to the enlargement ratio of enlargement processing inthe enlargement processing section 1.

[0050] The edge pattern selection section 13 comprises a function ofselecting a first edge pattern of the same size as the notable areacorresponding to the edge direction of the notable area estimated by theedge direction estimation section 11 and specifies a second edge patternof a different size from the first edge pattern, the first and secondedge patterns being provided in a one-to-one correspondence with eachother.

[0051] The enlarged image block generation section 14 comprises afunction of generating an enlarged image block for the notable areausing the edge direction estimated by the edge direction estimationsection 11, the second edge pattern provided by the edge patternselection section 13, and the contrast-enhanced pixel values provided bythe image enhancement section 12.

[0052] The image block placement section 15 comprises a function ofplacing the enlarged image blocks output from the enlarged image blockgeneration section 14 in order and outputting the image data subjectedto resolution conversion and enlargement to the image data storagesection 16.

[0053] Next, an outline of a flow of the enlargement processing in theembodiment of the invention will be discussed with reference to aflowchart of FIG. 2. In the description to follow, for referencenumerals not shown in FIG. 2, FIG. 1 is referenced.

[0054] To being with, at step S20, the image block setting section 10sets each block size required for processing of the edge directionestimation section 11 and the image enhancement section 12 for the inputimage data from the input unit (not shown) and stored the image datastorage section 16, and extracts the image block of the block size fromthe input image data.

[0055] If the notable area is of 2×2 size and the peripheral areacontaining the notable area is of 4×4 size, for example, as shown inFIG. 3, 6×6 size block shown in FIG. 3A is extracted for twiceenlargement; 8×8 size block shown in FIG. 3B is extracted for four-timeenlargement.

[0056] Next, at step S21, the edge direction estimation section 11calculates the edge directions of the notable area in the extractedimage block and the reference area in the peripheral area containing thenotable area and estimates edge direction Θ of the notable area from theprovided edge directions.

[0057] Next, at step S22, the edge pattern selection section 13 selectsfirst and second edge patterns using the edge direction Θ estimated bythe edge direction estimation section 11 and a pixel distributionpattern of the notable area. The first and second edge patterns areprovided for each edge pattern and each pixel distribution patterndescribed later, and are stored in a storage section (not shown in FIG.1), in a form such as table information.

[0058] Subsequently, as step S23, the image enhancement section 12enhances the image data in the notable area and its peripheral area inthe image block extracted by the image block setting section 10 usingthe kernel having the element values and the size responsive to theenlargement ratio.

[0059] Next, at step S24, the enlarged image block generation section 14generates an enlarged image block for the notable area using the edgedirection Θ in the notable area estimated by the edge directionestimation section 11, the edge pattern selected by the edge patternselection section 13, and the pixel values of the notable area and theperipheral area enhanced by the image enhancement section 12.

[0060] Then, at step S25, the image block placement section 15 placesthe enlarged image blocks for the notable area, generated by theenlarged image block generation section 14 in order according to apredetermined method described later and stores the placed image blocksin the image data storage section 16.

[0061] At step S26, whether or not generation of the enlarged image datato be output relative to the input image data is complete is determined.If the generation of the enlarged image data is not complete, theprocess is returned to step S20 and another image block is extracted andthe above-described processing is repeated. If the generation of theenlarged image data is complete, the enlarged image data is output andthe enlargement processing is terminated.

[0062] The outline and the operation of the image processing apparatusin the embodiment of the invention have been described. Then, the edgedirection estimation section 11, the image enhancement section 12, theedge pattern selection section 13, the enlarged image block generationsection 14, and the image block placement section 15 of the main part ofthe image processing apparatus will be discussed in detail.

[0063] Edge direction estimation of the edge direction estimationsection 11 of the embodiment will be discussed in detail by taking as anexample the case where the notable area is a 2×2 size area and theperipheral area containing the notable area is a 4×4 size block aspreviously shown in FIG. 3.

[0064]FIG. 4A is a drawing to show an example of the notable area andthe peripheral area. In FIG. 4, the notable area is pixels {a, b, c,d}={15, 177, 86, 203} surrounded by a frame and the numerals indicatethe pixel values. A flow of edge direction estimation processing of theedge direction estimation section 11 will be discussed with reference toa flowchart of FIG. 5.

[0065] To begin with, at step S50, the edge direction Θ in the notableblock surrounded by the thick frame in FIG. 4 is calculated according tothe following expression (1). The number of angle references, a variablefor counting the number of edge angle references used for edge directionestimation of the notable area at step S55 described later, is set to 1.

gx=(a+c−b−d)/2

gy=(a+b−c−d)/2

Θ=arctan(gy/gx)  (1)

[0066] For example, for the notable area {15, 104, 86, 203} specificallyas shown in FIG. 4A, gx=−103 and gy=−85 from expression (1) and the edgedirection Θ in the notable area shown in FIG. 4A becomes equal to−140.5°. Further, as shown in FIG. 4B, if the found edge direction Θ isnormalized at 22.5° (eight directions), direction 2 results whenΘ=−145.2°.

[0067] Next, at step S51, reference areas used for edge directionestimation are selected from within the peripheral area shown in FIG. 4Ain response to the edge direction Θ in the notable area calculated atstep S50.

[0068]FIGS. 6A to 6C shows examples of reference area selection based onthe edge direction in the notable area. For the notable area, thenormalized edge direction is direction 2 and thus four reference areasshown in FIG. 6C (four areas of up and down and left and right 2×2containing the notable area) are selected. Reference area selection isnot limited to that shown in FIG. 6. For example, in FIG. 6C, eightreference areas (eight areas of up and down and left and right andslanting 2×2 containing the notable area) may be used.

[0069] Next, at step S52, for each reference area selected at step S51,the edge direction Θ is calculated according to expression (1) as atstep S50.

[0070] Then, at step S53, a comparison is made between the edgedirection in the selected reference area calculated at step S52 and theedge direction in the notable area calculated at step S50.

[0071] If the difference between both the edge directions is smallerthan a preset threshold value Th, the number of angle references isincremented by one at step S54 and the process is transferred to stepS55. If the difference between both the edge directions is larger thanthe preset threshold value Th, it is determined that the reference areais not adequate for edge direction estimation, and the process istransferred to step S55.

[0072] Next, at step S55, whether or not edge direction calculation inall selected reference areas is complete is determined. If the edgedirection calculation is complete, the process is transferred to stepS56; if the edge direction calculation is not yet complete, steps S52 toS54 are repeated.

[0073] At step S56, the sum total of the edge direction in the notablearea and the edge directions in the reference areas determined to beadequate for edge direction estimation at step S53 is calculated, andthe average edge direction resulting from dividing the sum total of theedge directions by the number of angle references is adopted as theestimated edge direction in the notable area.

[0074] Thus, the edge direction in the notable area is estimated alsoconsidering the edge directions in the reference areas derived from theedge direction calculated only in the notable area, so that it is madepossible to conduct edge direction detection with high accuracy only byperforming easy calculation.

[0075] The case where the input image data is gray scale image in theedge direction estimation section 11 is described. However, thisinvention is not limited in use of a gray scale image. By way ofexample, a flow of edge direction estimation processing for an RGB colorspace color image will be discussed with reference to a flowchart ofFIG. 7.

[0076] To begin with, at step S70, using the above-mentioned expression(1) and the following expression (2), edge strength G is calculated foreach of RGB color space blocks in a notable area:

G=gx*gx+gy*gy  (2)

[0077] Next, at step S71, the maximum edge strength is selected fromamong the edge strengths of the RGB color space blocks calculatedaccording to expression (2) and the color space block corresponding tothe selected maximum edge strength is selected.

[0078] Next, at step S72, steps S50 to S56 of the edge directionestimation processing previously described with reference to FIG. 5 areexecuted in the color space block selected at step S71.

[0079] Then, at step S73, the estimated edge direction in the colorspace block of the maximum edge strength is adopted as the estimatededge direction in other color space blocks, and processing of the edgepattern selection section 13 and the enlarged image block generationsection 14 described later is performed for each color space block.

[0080] As steps S70 to S73 are executed, it is made possible to suppressthe image quality degradation causes such as a color shift in each edgepart of enlarged image data in a color image.

[0081] The embodiment has been described assuming that normalizationafter edge direction calculation using expression (1) in notable andreference areas is eight directions, but this invention is not limitedto the embodiment. If higher-accuracy edge directions are required,normalization may be conducted to 12 directions (15.0°), or 16directions (12.25°).

[0082] Next, the operation of the edge pattern selection section 13 willbe discussed in detail. The edge pattern selection section 13 uses anedge pattern table, for example, as shown in FIG. 9, to select the firstedge pattern corresponding to the edge direction in the notable areaestimated by the edge direction estimation section 11 and determine thesecond edge pattern corresponding to the selected first edge pattern.

[0083] Specifically, as shown in FIG. 4A, the notable area surrounded bythe frame and the peripheral area of the notable area are used in thedescription to follow. The estimated edge direction for the notable areashown in FIG. 4A is estimated to be direction 1 as shown in FIG. 8according to processing steps of the edge direction estimation section11 described above (sum total of edge directions in notable area andfour reference areas (−737.3)/number of angle references (5)=−147.5,normalized edge direction 1).

[0084] Pattern candidates corresponding to the edge pattern in thenotable area are selected out of the edge pattern table shown in FIG. 9in accordance with the estimated edge direction in the notable area(direction 1). In this case, four patterns of patterns 0 to 3 of thefirst edge pattern of direction 1 become candidates for the first edgepattern for the notable area shown in FIG. 4A.

[0085] Next, the edge pattern selection section 13 selects any one ofpatterns 0 to 3 as the first edge pattern for the notable area asdescribed below.

[0086]FIGS. 10A to 10E are drawings to specifically describe a method ofselecting one edge pattern from among the four edge pattern candidatesfor the notable area shown in FIG. 4A. To begin with, the first edgepattern candidates are converted into bit patterns as shown in FIG. 10B(white part is set to 0; otherwise 1). However, the edge pattern tableshown in FIG. 9 may be previously converted into a bit pattern table andthe bit pattern table may be stored in the storage section (not shown inFIG. 1), in which case the step can be skipped.

[0087] Next, according to the following expression (3), the averagepixel value in the notable area (in FIG. 4A, 102) is calculated, theaverage value is subtracted from each pixel value of the notable area,and a pixel value pattern of the notable area is prepared based on thesigns (FIG. 10C). Further, the pixel value pattern is converted into abit pattern (FIG. 10D).

Mean=(a+b+c+d)/4

a_sign=a−Mean

b_sign=b−Mean

c_sign=c−Mean

d_sign=d−Mean  (3)

[0088] Next, pattern matching is performed between the bit pattern ofeach edge pattern candidate in FIG. 10B and the bit pattern of thenotable area in FIG. 10D, and one first edge pattern for the notablearea is selected (FIG. 10E). In this case, pattern 2 is selected as thefirst edge pattern for the notable area.

[0089] When the first edge pattern is selected, last the second edgepattern provided corresponding to the first edge pattern is determined.In this case, the second edge pattern shown in FIG. 11 is selected. Thesecond edge pattern is used for generating an enlarged image block forthe notable area in the enlarged image block generation section 14described later.

[0090] The first and second edge patterns are not limited to those shownin FIG. 9. For example, edge patterns different from those shown in FIG.9 may be used in response to the type of input image data, and thenumber of first and second edge pattern candidates at each angle may beincreased or decreased.

[0091] Next, the image enhancement section 12 will be discussed indetail. The image enhancement section 12 uses the kernel of elements andthe size responsive to the enlargement ratio of enlargement processingin the enlargement processing section 1 to enhance the contrast of theimage data of the notable area and its peripheral area in the imageblock extracted by the image block setting section 10 as shown in FIG.3.

[0092]FIGS. 12A to 12C show specific examples of enhancement kernelsused with the image enhancement section 12. FIG. 13 is a drawing to showhow pixel P is enhanced using kernel 0 shown in FIG. 12A. In this case,pixel value P′ of the pixel P is calculated according to the followingexpression (4):

Notable pixel value P′=1.60*P−0.15*(a+b+c+d)  (4)

[0093] (wherein a, b, c, d, and P denote pixel values of positions shownin FIG. 13.)

[0094]FIG. 14 shows an example of contrast enhancement using theenhancement kernels shown in FIGS. 12A to 12C to enlarge input imagedata eight times. First, to enlarge the input image data twice, kernel 0shown in FIG. 12A is used to execute contrast enhancement of the inputimage data.

[0095] Next, to enlarge twice the input image data enlarged twice,kernel 1 shown in FIG. 12B is used to execute contrast enhancement ofthe input image data twice enlarged. Likewise, to enlarge the inputimage data eight times, kernel 2 shown in FIG. 12C is used to executecontrast enhancement of the input image data enlarged four times.

[0096] The kernels used for performing contrast enhancement are notlimited to those shown in FIGS. 12A to 12C, and a kernel different fromthose shown in FIGS. 12A to 12C in elements and element-to-elementdistance may be used in response to the type or size of input imagedata.

[0097] Next, the enlarged image block generation section 14 will bediscussed. The enlarged image block generation section 14 generates anenlarged image block for the notable area using the second edge patternprovided by the edge pattern selection section 13 and the pixel valuessubjected to contrast enhancement in the image enhancement section 12.

[0098] An example of a flow of enlarged image block generationprocessing of the enlarged image block generation section 14 will bediscussed with reference to a flowchart of FIG. 15. To begin with, atstep S150, a 3×3 enlarged image block is generated using the notablearea subjected to contrast enhancement in the image enhancement section12 and the second edge pattern selected by the edge pattern selectionsection 13, as shown in FIG. 16. The pixel values of the 3×3 enlargedimage block are calculated according to expressions shown in FIG. 16.The pixel value calculation expressions are determined for each secondedge pattern. FIG. 17 shows specific examples of the pixel valuecalculation expressions in other second edge patterns.

[0099] Next, at step S151, the edge direction in the notable areaestimated by the edge direction estimation section 11 is determined. Ifthe estimated edge direction is any of directions 1 to 3 or directions 5to 7, the process is transferred to step S152; if the estimated edgedirection is direction 0 or 4, the process is transferred to step S153.

[0100] At step S152 (when the estimated edge direction in the notablearea is any of directions 1 to 3 or directions 5 to 7), a 4×4 enlargedimage block is generated from the 3×3 enlarged image block generated atstep S150.

[0101] To begin with, as shown in FIG. 18, a 3×4 enlarged image block isgenerated using the 3×3 enlarged image block and reference pixels (r0 tor5) in the peripheral area subjected to contrast enhancement in theimage enhancement section 12. The pixel values of the 3×4 enlarged imageblock are determined according to calculation expressions shown in FIG.18. The reference pixels (r0 to r5) in the peripheral area are selectedin accordance with the estimated edge direction in the notable area.

[0102]FIG. 19 shows specific examples of the reference pixels selectedaccording to the estimated edge direction. Selection of the referencepixels is not limited to the selection examples of two patterns as shownin FIG. 19, and a larger number of reference pixel selection patternsmay be provided in accordance with the estimated edge direction.

[0103] Next, as shown in FIG. 20, a 4×4 enlarged image block isgenerated using the 3×4 enlarged image block and reference pixels (r0 tor7) in the peripheral area subjected to contrast enhancement in theimage enhancement section 12. The pixel values of the 4×4 enlarged imageblock are determined according to calculation expressions shown in FIG.20.

[0104] In the 4×4 enlarged image block generation at step S152 describedabove, the processing flow of 3×3 block -> 3×4 block -> 4×4 blockgeneration is shown, but a processing flow of 3×3 block -> 4×3 block ->4×4 block generation may be adopted, in which case reference pixelselection is appropriately changed.

[0105] At step S153 (when the estimated edge direction in the notablearea is direction 0 or 4), a 4×4 enlarged image block is generated fromthe 3×3 enlarged image block generated at step S150.

[0106]FIG. 21 shows an outline of 4×4 enlarged image block generationprocessing at step S153. First, the notable area and peripheral areablock (4×4) subjected to contrast enhancement in the image enhancementsection 12 is enlarged 1.25 times. In this case, the enlargementtechnique may be linear enlargement or projection enlargement.

[0107] Next, the center portion of the provided 5×5 block (3×3 block) isreplaced with the 3×3 enlarged image block generated at step S150 andfurther the resultant 5×5 block is enlarged 1.2 times. Also in thiscase, the enlargement technique may be linear enlargement or projectionenlargement.

[0108] Last, the center portion of the provided 6×6 block (4×4 block) isextracted as a 4×4 enlarged image block for the notable area. As stepsS150 to S153 are executed, the enlarged image block for the notablearea, for example, as shown in FIG. 22 is generated.

[0109] Next, the image block placement section 15 will be discussed. Theimage block placement section 15 places the enlarged image blocks forthe notable area generated by the enlarged image block generationsection 14 in order according to a predetermined method.

[0110]FIG. 23 shows a specific example of placing the 4×4 enlarged imageblocks generated by the enlarged image block generation section 14. Inthe example shown in FIG. 23, enlarged image blocks 0 and 1 generated inorder are placed so that they overlap each other. Each overlapped pixelis placed in such a manner that the pixel value and the preceding pixelvalue are averaged.

[0111] To perform image enlargement processing by the image processingapparatus and the image processing method of the embodiment, the edgeshape pattern corresponding to the edge direction can be selected onlyby executing simple pattern matching and an enlarged image is generatedusing the pixel values considering the edge direction based on theselected edge shape pattern, so that it is made possible to provide anenlarged image with occurrence of blurring and jaggies suppressed undersmall processing load.

[0112] The image processing method described above may be executed in apersonal computer or a digital camera, as an image processing program ormay be a form distributed via a communication line of the Internet, orrecorded on a computer-readable record medium, such as a CD-ROM.

[0113] Such an image processing program can also be applied to the casewhere an input image is enlarged digitally in a machine for handling adigital image, such as a digital still camera, a digital video camera, amobile telephone, or a PDA (personal data assistant).

[0114] As described above, in the invention, in image enlargementprocessing, a precise edge direction is detected and an enlarged imageis generated from the pixel values responsive to the edge direction, sothat it is made possible to accomplish high-quality image enlargementprocessing with image quality defects of blurring and jaggies suppressedunder small processing load at high speed.

[0115] Although the present invention has been shown and described withreference to specific preferred embodiments, various changes andmodifications will be apparent to those skilled in the art from theteachings herein. Such changes and modifications as are obvious aredeemed to come within the spirit, scope and contemplation of theinvention as defined in the appended claims.

What is claimed is:
 1. An image processing apparatus for performingimage enlargement processing, said image processing apparatuscomprising: an edge direction estimation section for estimating edgedirection information for a first image area containing a notable pixel;an edge pattern selection section for selecting an edge shape patterncorresponding to the first image area corresponding to an edge directionindicated by the edge direction information based on the edge directioninformation and a pixel value in the first image area; an imageenhancement section for enhancing the pixel value of the first imagearea; an enlarged image area generation section for generating anenlarged image area by using the edge shape pattern selected by saidedge pattern selection section and the pixel value of the first imagearea enhanced by said image enhancement means; and an image placementsection for placing the enlarged image area generated by said enlargedimage area generation section according to a predetermined method. 2.The image processing apparatus as claimed in claim 1, wherein said edgedirection estimation section calculates each edge angle based on thepixel value difference for the first image area containing the notablepixel and an image area on the periphery of the first image area, andestimates the edge direction information in the first image areacontaining the notable pixel from the calculated edge angles.
 3. Theimage processing apparatus as claimed in claim 1, wherein said edgedirection estimation section selects the image area used for estimatingthe edge direction information in the first image area containing thenotable pixel.
 4. The image processing apparatus as claimed in claim 1,wherein the edge direction information estimated by said edge directionestimation section is normalized in a determined number of directions.5. The image processing apparatus as claimed in claim 1, wherein saidedge direction estimation section calculates edge strength informationin the first image area.
 6. The image processing apparatus as claimed inclaim 5, wherein said edge direction estimation section selects a colorspace data piece from among color space data pieces in accordance withthe edge strength information and estimates the edge directioninformation using the selected color space data piece.
 7. The imageprocessing apparatus as claimed in claim 1, wherein said edge patternselection section selects the edge shape pattern from among a pluralityof edge shape patterns for specific edge direction information.
 8. Theimage processing apparatus as claimed in claim 1, wherein when two ormore edge shape patterns are selectable for specific edge directioninformation, said edge pattern selection section converts the pixels inthe first image area into a binary pattern and executes pattern matchingbetween the binary pattern and each of the edge shape patterns to selectthe edge shape pattern.
 9. The image processing apparatus as claimed inclaim 1, wherein said edge pattern selection section selects the edgeshape pattern from among a plurality of edge shape patterns different insize corresponding to the first image area, each edge shape patternbeing provided in a correspondence with different size.
 10. The imageprocessing apparatus as claimed in claim 1, wherein said imageenhancement section selects a enhancement kernel in response to anenlargement ratio from among a plurality of enhancement kernelsdifferent in kernel elements and element-to-element distance and appliesthe selected enhancement kernel to enhancing the pixel values of thefirst image area.
 11. The image processing apparatus as claimed in claim1, wherein said enlarged image area generation section comprises firstenlarged image area generation section and second enlarged image areageneration section.
 12. The image processing apparatus as claimed inclaim 11, wherein the first enlarged image area generation sectionenlarges the first image area to generate a first enlarged image areafor each of the edge shape patterns by using the pixel value in thefirst image area enhanced by said image enhancement section or by usinga value calculated using two or more pixel values in the first imagearea enhanced by said image enhancement section.
 13. The imageprocessing apparatus as claimed in claim 11, wherein the second enlargedimage area generation section comprises a plurality of enlargement unitthat enlarges the first enlarged image area by using different process;and the second enlarged image area generation section selects aenlargement unit to use from among the plurality of enlargement unit inaccordance with the edge direction information estimated by said edgedirection estimation section.
 14. The image processing apparatus asclaimed in claim 13, wherein one of the enlargement unit generates asecond enlarged image area by predetermined calculation using the pixelvalue in the first enlarged image area and the pixel value in theperiphery of the first image area enhanced by said image enhancementsection.
 15. The image processing apparatus as claimed in claim 13,wherein one of the enlargement unit selects the pixel value in theperiphery of the first image area in accordance with the edge directioninformation estimated by said edge direction estimation section.
 16. Theimage processing apparatus as claimed in claim 13, wherein one of theenlargement unit generates a second enlarged image area using a valueresulting from converting the pixel value in the periphery of the firstimage area enhanced by said image enhancement section by predeterminedcalculation.
 17. The image processing apparatus as claimed in claim 1,wherein said image placement section places the enlarged image areasgenerated by said enlarged image area generation section sequentially soas to overlap the each enlarged image areas.
 18. An image processingapparatus comprising: an image data obtaining means for obtaining animage data; a pattern data storing means for storing a pattern data; andan image replacing means for replacing an area image data correspondingto a specific area of an image contained in the image data and havingsmaller area size than the area image data with the pattern data havingsimilarity to the area image data.
 19. An image processing method forperforming image enlargement processing, said image processing methodcomprising the steps of: estimating edge direction information for afirst image area containing a notable pixel; selecting an edge shapepattern corresponding to the first image area corresponding to an edgedirection indicated by the edge direction information based on the edgedirection information and a pixel value in the first image area;enhancing the pixel value of the first image area; generating anenlarged image area by using the edge shape pattern and the enhancedpixel value of the first image area; and placing the enlarged image areaaccording to a predetermined method.
 20. An image processing program forperforming image enlargement processing, said image processing programcomprising the steps of: estimating edge direction information for afirst image area containing a notable pixel; selecting an edge shapepattern corresponding to the first image area corresponding to an edgedirection indicated by the edge direction information based on the edgedirection information and a pixel value in the first image area;enhancing the pixel value of the first image area; generating anenlarged image area by using the edge shape pattern and the enhancedpixel value of the first image area; and placing the enlarged image areaaccording to a predetermined method.
 21. A computer-readable recordmedium recording an image processing program comprising the steps of:estimating edge direction information for a first image area containinga notable pixel; selecting an edge shape pattern corresponding to thefirst image area corresponding to an edge direction indicated by theedge direction information based on the edge direction information and apixel value in the first image area; enhancing the pixel value of thefirst image area; generating an enlarged image area by using the edgeshape pattern and the enhanced pixel value of the first image area; andplacing the enlarged image area according to a predetermined method.